TITLE:  Look  and  Feel:  Haptic  Interaction  for  Biomedicine 


PRINCIPAL  INVESTIGATOR:  Marc  Raibert 


CONTRACTING  ORGANIZATION:  Boston  Dynamics,  Incorporated 

Cambridge,  Massachusetts  02139 


REPORT  DATE:  November  1997 

Reproduced  From 
Best  Available  Copy 

TYPE  OF  REPORT:  Final 


PREPARED  FOR:  U.S.  Army  Medical  Research  and  Materiel  Command 

Fort  Detrick,  Maryland  21702-5012 


DISTRIBUTION  STATEMENT:  Approved  for  public  release; 

distribution  unlimited 


The  views,  opinions  and/or  findings  contained  in  this  report  are 
those  of  the  author (s)  and  should  not  be  construed  as  an  official 
Department  of  the  Army  position,  policy  or  decision  unless  so 
designated  by  other  documentation. 


REPORT  DOCUMENTATION  PAGE 

Form  Approvad 

OMd  No.  0704-0188 

PuMoraparting  burdjnfdr  t.ita  caHaetian  «f  Intarmadanta  aad mataa  ta  avr;^a  Ihata  par  niaanaa,  Inajudlng  tha  tima  far  rartawlng  Inaoucttana,  aaaraWng  aidaCna  im.  i  aavoaa, 
gathering  and  maintaining  tha  d  ita  naadad,  and  aampfatlno  and  raviawlng  tha  oaNaatlan  af  inf armadan.  Sand  oammama  ragarding  tfaa  btadan  aadmata  ar  any  athar  aapaa '  af  thia 
cadaedan at  tnfarmadan,  including  auggaadana  far  raduglng  dda  burdan,  ta  WaaNngtan  Haadguartara  Baryte  \  Direct jrata far Irfarmatlan  Ogaradana  and fUgarta,  111*  Jvdaraan 

Davia  Highway,  Stdta  1204,  Arlington,  VA  22202-4302,  and  ta  tha  Offlaa  af  Management  and  Budget,  P^rwetfc  Haduadan  Prelect  10704-01  SSI,  WaaNngtan,  DC  20 403. 

1 .  AGENCY  USE  ONLY  (Laava  blank) 

2.  REPORT  DATE 

November  1997 

3.  REPORT  TYPE  AND  DATES  COVERED 

Final  (30  Sep  94  -  29  Oct  97) 

U.  TITLE  AND  SUBTITLE 

5.  FUNDING  NUMBERS  1 

Look  and  Feel:  Haptic  Interaction  for  Biomedicine 


DAMD17-94-C-4124 


6.  AUTHOR(S) 


Raibert,  Marc 


7.  PERFORMING  ORGANIZATION  NAME(S)  AND  AODRESS(ES) 

Boston  Dynamics,  Incorporated 

Cambridge,  Massachusetts  02139 

8.  PERFORMING  ORGANIZATION 

REPORT  NUMBER 

9.  SPONSORING  /  MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 

U.S.  Army  Medical  Research  and  Materiel  Command 

Fort  Detrick,  Maryland  21702-5012 

10.  SPONSORING  /  MONITORING 

AGENCY  REPORT  NUMBER 

1 1 .  SUPPLEMENTARY  NOTES 

Appendices  to  Final  report  is  film  entitled  "Virtual  Surgery" 
dated  September  1997. 

12a.  DISTRIBUTION  /  AVAILABILITY  STATEMENT 

Approved  for  public  release;  distribution  unlimited 

12b.  DISTRIBUTION  CODE 

13.  ABSTRACT  (Maximum  200  words) 

Our  goal  is  to  develop  the  technological  foundation  of  advanced  haptic  systems  and  to  assemble  that  technology  into  a  useful 
surgical  simulator.  The  surgical  simulator  we  envision  will  allow  training  of  biomedical  procedures  in  a  virtual  environment 
that  features  realistic  interaction  with  simulated  anatomy  and  surgical  tools  through  haptic  force  feedback,  3D  graphics,  and 
sound.  In  addition  to  direct  training,  surgical  emulators  could  be  used  to  assess  surgical  performance  and  play  a  role  in 
certification.  This  report  describes  our  progress  in  building  such  a  system. 

A 


\ 


14.  SUBJECT  TERMS 

Biomedicine,  Haptics  Applications,  Virtual  Surgery  Simulator 

15.  NUMBER  OF  PAGES 

25 

18.  PRICE  CODE 

17.  SECURITY  CLASSIFICATION 

18.  SECURITY  CLASSIFICATION 

19.  SECURITY  CLASSIFICATION 

20.  LIMITATION  OF  ABSTRACT 

OF  REPORT 

OF  THIS  PAGE 

OF  ABSTRACT 

Unclassified 

Unclassified 

Unclassified 

Unlimited 

NSN  7540  01-280-5500 


Standard  Form  298  (Rov.  2-89) 

Praacribed  by  ANSI  Std.  239-18  298-102 


usappc  vi  .00 


EQREWCRD 


Opinions/  interpretations,  conclusions  and  recommendations  are 
those  of  the  author  and  are  not  necessarily  endorsed  by  the  U.S. 
Army. 


_  Where  copyrighted  material  is  quoted,  permission  has  been 

obtained  to  use  such  material. 

_  Where  material  from  documents  designated  for  limited 

distribution  is  quoted,  permission  has  been  obtained  to  use  the 
material. 

_  Citations  of  commercial  organizations  and  trade  names  in 

this  report  do  not  constitute  an  official  Department  of  Army 
endorsement  or  approval  of  the  products  or  services  of  these 
organizations . 

_  In  conducting  research  using  animals,  the  investigator (s) 

adhered  to  the  "Guide  for  the  Care  and  Use  of  Laboratory 
Animals,"  prepared  by  the  Committee  on  Care  and  Use  of  Laboratory 
Animals  of  the  Institute  of  Laboratory  Resources,  National 
Research  Council  (NIH  Publication  No.  86-23,  Revised  1985). 

_  For  the  protection  of  human  subjects,  the  investigator (s) 

adhered  to  policies  of  applicable  Federal  Law  45  CFR  46. 

_  In  conducting  research  utilizing  recombinant  DNA  technology, 

the  investigator (s)  adhered  to  current  guidelines  promulgated  by 
the  National  Institutes  of  Health. 

_  In  the  conduct  of  research  utilizing  recombinant  DNA,  the 

investigator (s)  adhered  to  the  NIH  Guidelines  for  Research 
Involving  Recombinant  DNA  Molecules. 

_  In  the  conduct  of  research  involving  hazardous  organisms, 

the  investigator (s)  adhered  to  the  CDC-NIH  Guide  for  Biosafety  in 
Microbiological  and  Biomedical  Laboratories. 


February  2, 1998 


PI  -  Signature 


Date 


Introduction 

Integrated  Haptics  Applications 

The  Virtual  Surgery  Simulator 

Performance  Assessment  with  the  VR  Surgical  Trainer 

Palpation  of  a  Beating  Heart 

Progress  on  Components  of  an  Advanced  Haptic  System 
Haptic  Algorithms 

Contact  Detection 
Contact  Force  Models 
Haptic  Texture  Mapping 
Physics-Based  Simulation 
Rigid  Object  Loader 
Flexible  Tissue 

Commercialization  and  Related  Work 
Conclusions 

APPENDIX  A  -  A  Novel  Virtual  Reality  Surgical  Trainer  with  Force  Feedback: 
Surgeon  Vs.  Medical  Student  Performance 


APPENDIX  B  -  A  Virtual  Surgery  Simulator  Using  Advanced  Haptic  Feedback 


Look  and  Feel 


Haptic  Interaction  for  Biomedicine 

1997  Final  Report 


Introduction 


Figure  1:  Photograph  of  trainee  using 
BD1  Surgical  Simulator,  Computer 
graphics  creates  3D  views  of  the  virtual 
organs  which  are  presented 
stereoscopically,  Real  surgical  tools  are 
attached  to  force  feedback  devices  to 
measure  the  trainee's  motions  and  feed 
back  forces  to  the  hands.  Physics-based 
simulations  mediate  interactions 
between  tools  and  organs. 


People  use  their  hands  to  touch,  manipulate,  and  learn  about  the  world  around  them.  Interaction  through 
touch  and  manipulation  is  known  as  haptic  interaction.  We  are  working  to  develop  advanced  haptic 
technology  that  allows  human  users  to  touch,  feel,  grasp,  and  manipulate  a  set  of  special  objects:  objects 
located  remotely,  objects  too  smal1  too  large,  or  too  dangerous  for  normal  human  interaction,  or  virtual 
objects  that  exist  only  in  simulated  worlds. 

Advanced  haptic  interaction  will  have  a  wide  range  of  applications  in  biomedical,  military,  and  civilian  areas. 
It  will  allow  design  engineers  to  "pick  up"  new  designs,  "try  them  out,"  and  see  how  they  behave,  before 
they  ever  leave  the  drawing  board.  Engineers  will  sculpt  the  shapes  of  new  parts  in  the  computer,  as  though 
made  of  clay.  Remote  experts  will  aid  the  local  novice  in  performing  delicate  procedures.  Medical  students 
will  train  by  manipulating  virtual  organs,  without  risk  to  human  patients  or  animals.  Students  will  probe  any 
part  of  the  virtual  patient  with  impunity  and  view  a  wide  range  of  conditions  from  any  vantage  point.  This 
work  will  enable  applications  ranging  from  familiarization  and  training  in  domains  requiring  hand/eye  skills 
(surgery,  mechanics,  maintenance,  for  example)  to  the  enhancement  of  operations  in  remote,  scaled  , 
unfamiliar,  and  hazardous  environments. 

Our  goal  is  to  develop  the  technological  foundation  of  advanced  haptic  systems  and  to  assemble  that 
technology  into  a  useful  surgical  simulator.  The  surgical  simulator  we  envision  will  allow  training  of 
biomedical  procedures  in  a  virtual  environment  that  features  realistic  interaction  with  simulated  anatomy  and 
surgical  tools  through  haptic  force  feedback,  3D  graphics,  and  sound.  In  addition  to  direct  training,  surgical 
emulators  could  be  used  to  assess  surgical  performance  and  play  a  role  in  certification.  This  report  describes 
our  progress  in  building  such  a  system. 
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In  the  first  year  of  this  effort  we  focused  on  developing  the  basic  technology  and  methods  needed  to  build 
interactive  haptic  applications.  In  the  second  year  of  effort,  we  have  assembled  these  components  into 
several  prototype  systems.  We  have  built  integrated  systems  for  surgical  anastomosis,  heart  palpation  and 
aircraft  maintenance.  These  systems  were  built  upon  a  common  software  foundation  that  includes  modules 
for  virtual  haptic  interaction,  physics-based  simulation,  and  computer  graphics.  In  the  last  year  of  effort,  we 
extended  the  simulator  to  accommodate  measurement  of  surgical  performance.  We  used  the  simulator  in  a 
preliminary  test  comparing  the  performance  of  medical  students  and  experienced  vascular  surgeons. 

Integrated  Haptics  Applications 

Our  advanced  haptic  systems  consist  of  a  mechanical  force-feedback  device,  a  real-time  computing  system,  a 
dynamic  simulation  system,  and  computer  graphics  and  sound  (Figure  2).  Next  we  describe  these  systems:  a 
simulator  for  surgical  anastomosis,  a  version  of  the  anastomosis  simulator  that  includes  performance 
assessment  software,  demonstration  of  virtual  heart  palpation  that  uses  volumetric  3D  visual  displays,  and  a 
simulator  for  aircraft  maintenance.  While  the  functionality  of  each  of  these  systems  is  significantly  different, 
they  all  build  upon  a  common  foundation  and  thus  contribute  to  the  overall  goal  of  developing  advanced 
haptic  systems. 


Figure  2:  Haptic,  Graphic,  and  Audio  Presentation  of  a  Virtual  Object 


The  Virtual  Surgery  Simulator 

The  primary  focus  of  activity  of  the  "Look  and  Feel"  project  remains  the  development  of  the  Virtual  Surgery 
Simulator  (Figure  1).  This  simulator  is  an  integrated  system  that  allows  users  to  reach  into  a  virtual  human 
body  to  touch,  feel,  grasp  and  suture  two  deformable  tubes.  The  user  boids  real  surgical  tools  connected  to 
Phantom  force  feedback  devices.  By  holding  a  needle  holder  in  one  hand  and  forceps  in  the  other,  the  user 
can  manipulate  two  virtual  tube  organs  and  suture  them  together  to  practice  an  end-to-end  anastomosis.  The 
forces  of  interaction  between  the  tools  and  the  simulated  tubes  are  reflected  to  the  user  with  the  Phantom 
while  the  virtual  workspace  is  visually  displayed  using  3Dcomputer  graphic  images.  Coordinated  visual  and 
tactile  displays  allow  the  user  to  work  in  an  intuitive  way  while  exploring  the  basic  techniques  of  the  surgical 
anastomosis. 
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Figure  3:  The  I  'irlual  Surgery  Simulator  features  deformable  tubes  that  can  be  sewn  together  using  needle, 
suture,  needle  holder,  and  forceps. 


The  simulated  elements  in  the  anastomosis  demonstration  are  two  deformable  tubes,  forceps,  needle  holder, 
needle  and  suture.  The  look  and  feel  of  the  tubes  can  be  changed  to  represent  different  anatomical  elements. 
Tube  diameter,  thickness,  length,  compliance,  surface  friction,  and  resistance  to  puncture  are  just  a  few 
parameters  that  can  be  used  to  accommodate  different  surgical  applications.  Texture  mapped  graphics 
derived  from  photographs  of  real  biological  tissue  help  make  the  simulated  elements  look  realistic. 

We  have  constructed  a  Virtual  Surgery  Station  that  allows  the  surgeon  and  surgical  subject  to  be  in  the 
correct  relative  position  (Figure  4).  The  user  stands  with  hands  at  table  height  holding  the  forceps  and  the 
needle  holder.  Looking  down  at  his  hands,  the  surgeon  instead  sees  a  mirror  reflecting  the  computer  graphic 
image  of  the  subject  and  the  virtual  tools.  This  image  is  produced  in  approximately  the  correct  location  by 
using  an  SGI  monitor  mounted  overhead.  A  hand  rest  allows  the  surgeon  to  stabilize  his  hands  for  delicate 
manipulation.  We  have  delivered  a  version  of  this  Virtual  Surgery  Station  to  Dr.  Tom  Krummel  at 
Pennsylvania  State's  Hershey  Medical  Center.  Dr.  Krummel  intends  to  integrate  this  trainer  into  his  virtual 
training  lab  at  the  Hershey  Medical  Center.  We  have  also  demonstrated  this  system  at  the  meeting  of  the 
Association  of  Academic  Health  Centers,  the  society  for  Vascular  Surgeons  Conference  and  the  American 
College  of  Surgeons  Conference  . 
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Figure  4:  The  virtual  reality  surgical  trainer  lets  you  see,  tou>.  ,  and  feel  simulated  organs  using  3D  computer 
graphics  and  advanced  force  feedback  technology.  Using  a  needle  holder  and  forceps  attached  to  force 
feedback  devices,  the  user  can  grasp,  poke,  pluck,  and  suture  flexible  tube  organs.  The  user  can  feel  the 
vessels  when  touched  and  the  vessels  move  realistically  in  response  to  touch.  The  surgical  station  uses  a 
mirror  arrangement  to  place  the  3D  image  of  the  patient  in  the  correct  position  relative  to  the  user.  This  system 
allows  the  user  to  control  a  virtual  needle  and  thread  to  perform  a  simulated  end-to-end  anastomosis. 

Performance  Assessment  with  the  VR  Surgical  Trainer 

The  virtual  reality  (VR)  surgical  trainer  is  a  modification  of  the  simulator  described  above  that  assesses  skill 
level  during  a  simulated  surgical  procedure.  The  virtual  surgical  performance  is  evaluated  based  upon 
measures  of  damage  to  the  tissue,  needle  technique,  and  other  dynamic  parameters.  An  initial  experiment  has 
compared  the  performance  of  surgeons  and  medical  students  on  a  simulated  surgical  task.  We  found  that  on 
average  the  surgeons  performed  better  than  the  medical  students. 

Virtual  reality  simulations  hold  the  potential  to  improve  surgical  training  in  a  manner  analogous  to  its  impact 
on  the  training  of  pilots.  The  advantages  of  VR  for  surgical  training  include  a  reduction  in  operating  room 
time  required  for  training,  the  quantification  of  trainees’  performances,  and  an  ability  to  provide  experiences 
independent  of  a  hospital's  limited  patient  population.  An  interactive  virtual  reality  surgical  trainer  could 
improve  the  training  of  surgeons  and  also  play  a  role  in  the  accreditation  process. 


We  conducted  an  experiment  to  compare  the  performance  of  medical  students  and  vascular  surgeons  using 
the  VR  surgical  trainer.  Both  groups  were  given  the  task  of  repeatedly  inserting  a  curved  needle  through  a 
simulated  vessel,  Twelve  Harvard  Medical  School  students  and  nine  vascular  surgeons  from  five  Boston  area 
hospitals  participated  in  the  study.  We  tracked  eight  parameters  during  the  study:  1)  tissue  damage,  2) 
accuracy,  3)  peak  force  applied  to  the  tissue,  4)  time  to  complete  the  task,  3)  surface  damage  to  the  tissue,  6) 
total  tool  movement.  7)  average  angular  error  from  an  ideal  needle  path,  and  8)  an  overall  error  score.  The 
users  were  given  visual  feedback  regarding  their  performance  after  each  trial  of  the  experiment. 

The  surgeons  tended  to  outperform  the  medical  students  by  having  less  tissue  damage  (See  for  example 
Figure  4).  The  surgeons'  average  performance  scores  were  better  than  those  of  the  medical  students  for  all  of 
the  measured  parameters  (See  Figure  5).  These  differences  were  statistically  significant  (p<0.05)  for: 
damage  to  the  tissue,  time  to  complete  the  task,  excessive  tool  motions,  proper  angle  of  the  needle,  and 
overall  error  score. 


Average  Tissue  Damage  vs.  Trial  Number 


Figure  S:  The  average  tissue  damage  values  are  plotted  versus  the  trial  number  in  the  experiment.  Tissue 
damage  is  defined  as  the  sum  of  forces  that  exceed  a  tissue  damage  threshold. 
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Virtual  Reality  Surgical  Performance: 
Medical  Studenta  va.  Surgeons 


Par  torn  ance  Ermr  Score 
Tissue  Damage 
Sir  (ace  Damage 
Excess  Time 
Accuracy  Error 
Excess  Tool  Motion 
Angle  Error 

Peak  Force 
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Average  Perforamnce 
(Normalized  to  Surgeons'  Average  Score) 


Figure  6:  The  bar  churl  displays  the  average  medical  student  and  surgeon  performances  for  the  simulated  surgical 
task  The  values  have  been  normalized  to  the  average  surgeon  values.  The  surgeon  values  were  better  than  the 
medical  student  values  for  all  eight  of  the  metrics.  Five  of  the  differences  ( performance  error  score,  tissue  damage, 
excess  time,  excess  tool  motion,  and  angle  error)  were  statistically  significant  (p<0.05). 


We  have  developed  a  VR  trainer  that  tests  a  user’s  technique  in  placing  a  needle  through  a  simulated  vessel. 
An  initial  experiment  has  compared  the  simulated  surgical  performance  of  medical  students  and  vascular 
surgeons.  On  average,  the  surgeons  outperformed  the  medical  students  in  the  8  parameters  that  were 
measured  (statistically  significant  for  5  of  the  parameters).  These  results  suggest  that  the  VR  surgical  trainer 
may  be  useful  in  quantifying  surgical  skill. 


Palpation  of  a  Beating  Heart 

Boston  Dynamics  Inc,  in  conjunction  with  Digital  Media  Associates  has  created  a  system  that  allows  the  user 
to  touch,  poke  and  pluck  a  simulated,  beating  heart  with  medical  instruments  (Figure  6).  The  user  can  touch 
a  3D  volumetric  display  of  the  heart  with  medical  instruments  mounted  to  Phantom  force-feedback  devices. 
The  user  actually  feels  the  heart  whenever  the  physical  tool  "touches"  the  floating  3D  image  because  the 
visual  and  haptic  images  of  the  heart  are  coincident  in  space.  An  audible  heart  beat  is  synchronized  with  the 
visual  and  haptic  heart  displays. 
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Figure  7:  The  I  'irtual  Heart:  the  user  can  see,  feel,  and  hear  the  heart  heat  as  well  as  poke  and 
pluck  its  deformable  surface 


The  volumetric  3D  display  is  created  by  projecting  computer  graphic  images  of  a  beating  heart  onto  a  screen 
contoured  to  the  shape  of  a  heart.  The  image  of  this  screen  is  projected  into  the  haptic  workspace  using 
mirrors.  The  projection  system,  the  3D-RT,  was  built  by  Digital  Media  Associates,  it  consists  of  a  high 
quality  projector,  the  heart  contoured  screen,  and  the  mirroring  system.  BDI  created  the  haptic  algorithms, 
dynamic  simulation,  computer  graphics,  and  sound  of  the  beating,  deformable  heart.  This  unique  combined 
display  system  makes  it  easier  to  touch  virtual  objects  by  puttii  g  haptic  and  visual  images  together. 

Virtual  Aircraft  Maintenance  Training 

The  technology  that  provides  the  foundation  for  advanced  haptics  systems  is  largely  independent  of  the 
application  area.  In  a  separately  funded  project  we  developed  the  Virtual  Aircraft  Maintenance  Trainer 
(Figure  3).  This  project  has  leveraged  the  funding  for  surgical  simulator  development  because  the  basic 
issues  regarding  dynamic  simulation,  haptic  force- feedback,  and  computer  graphics  are  common  to  all 
haptics  systems.  We  developed  the  Virtual  Aircraft  Maintenance  Trainer  to  explore  the  feasibility  of 
replacing  expensive  mock-ups  of  aircraft  used  for  maintenance  training  with  less  expensive  computer 
hardware  and  software.  The  virtual  main'  nance  trainer  we  built  allows  a  user  to  perform  the  essential 
elements  of  two  real-world  maintenance  training  tasks  performed  by  the  US  Marines  on  thcAV8B  aircraft,  a 
vertical  take-off  and  landing  attack  fighter.  One  task  is  diagnosis  of  a  failed  Radar  Warning  Receiver  System 
Built-In-Test  (RWR  BIT). The  second  maintenance  task  is  adjustment  of  the  Vemicr/Non-Linear  Nosewheel 
Steering  Linkage. 


Figure  H:  The  I  irnuil  Maintenance  Trainer  allows  the  user  to  i>ush  buttons,  throw  toggle  switches,  turn  dials, 
observe  lights,  and  listen  to  audible  tones,  and  manipulate  linkages  during  the  evaluation  and  adju:  Intent  of 
aircraft  systems 


Using  the  Virtual  Maintenance  Trainer  the  trainee  can  push  buttons,  throw  toggle  switches,  and  turn  dials  in 
the  cockpit,  as  well  as  observe  indicator  lights,  listen  for  audible  tones  and  measure  voltages  as  part  of 
running  and  diagnosing  an  avionics  self-test.  Simulations  of  the  cockpit  panels,  buttons,  dials,  switches,  and 
electrical  pins  allow  the  technician  to  see  feel,  and  hear  the  avionics  system  operations.  The  linkage 
adjustment  procedure  involves  manipulating  a  simulated  four-bar  linkage  through  its  range  of  motion, 
measuring  the  forces  required  to  move  the  linkage  between  two  hard  stops,  and  adjusting  those  forces  by 
tightening  or  loosening  a  friction  nut  on  the  linkage,  The  dynamic  simulation  of  the  linkage  obeys  the 
kinematic  and  dynamic  constraints  of  the  real  physical  device  and  allows  interactive  attachment  or 
detachment  from  the  rudder  pedal  actuator,  and  joint  friction  adjustment.  The  user  can  feel  the  shapes  of  the 
links  in  the  mechanism,  as  well  as  the  forces  needed  to  move  it. 


Progress  on  Components  of  an  Advanced  Haptic  System 

An  integrated  system  for  advanced  haptic  interaction  needs  to  have  three  parts:  a  part  that  mechanically 
interfaces  to  the  user's  hand  (and'or  arm),  a  part  that  mediates  the  interaction  between  the  hand  and  virtual  or 
remotely  located  objects,  and  a  part  that  simulates  how  the  objects  w  ill  behave  when  touched.  These 
components  are  force  feedback  devices,  haptic  algorithms,  and  physics-based  simulation,  respectively.  In 
Table  I  we  summarize  our  progression  a  list  of  development  tasks  for  advanced  haptic  systems  including  the 
virtual  surgical  simulator.  Next  we  detail  our  progress  in  these  areas. 
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Table  I:  Progress  List  for  Advanced  Haptic  System  Development 

Advanced  Hnptic  Interface  Devices :  The  haptic  device  is  the  mechanical  interface  that  allows  the  user 
to  touch  and  manipulate  a  virtual  or  remotely  located  object. 

•  Phantom  haptic  device  interfaced  to  SGI  and  PC 

•  Instrumented  medical  tool  attr  aments 

Debakey  forceps 
needle  holder 

•  Virtual  medical  tools 

debakey  forceps 

needle,  needle  holder,  and  suture 

scalpel  with  scalpel  holder 

hemostatic  forceps 

intestinal  forceps 

dissecting  forceps 

scissors 

retractor 

•  Instrumented  gimbal  measures  tool  position  and  orientation 

•  Powered,  two  finger  attachment  designed  and  tested  (MIT  Task) 

Haptic  Algorithms:  These  algorithms  mediate  the  forces  the  user  feels  with  the  Haptic  Interface  Device. 

•  Two  handed  interaction  demonstrated 
»  Two  finger  manipulation  demonstrated 

•  Rigid  body  contact  force  models 

•  Flexible  body  contact  force  models 

•  Coulomb  friction 

•  Haptic  textures  for  bumps,  sand  paper,  ridges 

•  Haptic  textures  mapped  onto  shapes 

•  Contact  detection  between  3D  polygonal  shapes 

•  Contact  defection  between  point  or  line  and  flexible  tubes 

•  Contact  detection  between  point  and  grid  maps 

Physics-Based  Simulation:  Real-time  physics-based  simulation  is  used  to  calculate  the  behavior  of  objects, 
interactions  among  objects,  and  interactions  between  objects  and  the  user. 

•  Automatic  simulation  generation  for  rigid  body  systems 

•  Loadable  object  shapes  from  CAD  files 

•  Programmable  bulk  properties  of  simulated  objects 

•  Compliant  flexible  skin 

•  Compliant  flexible  tube 

•  Calcified  artery 

•  Compliant  beating  heart 

•  Coordinated  3D  volumetric  display  and  haptic  display 

•  Suture  cwo  tubes  together  with  surgical  needle  and  suture 

•  Texture  mapped  graphics 
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Advanced  Haptic  Interface  Devices 

The  Phantom  haptic  device  is  a  light  weight,  high  bandwidth  mechanical  interface  that  allows  the  user  to 
touch  and  manipulate  a  virtual  or  remotely  located  object.  Haptic  fidelity  and  high  resolution  are  important 
for  conveying  a  detailed  sense  of  touch  and  are  thus  emphasized  over  kinematic  complexity  and  high  force 
capacity.  High  fidelity  and  resolution  are  obtained  by  maximizing  bandwidth  through  the  design  of  stiff, 
lightweight,  and  low  friction  structures  and  transmissions. 

The  Phantom  is  interfaced  to  both  a  PC  and  an  SGI  graphics  workstation.  Haptic  algorithms  and  object 
simulation  code  are  run  on  the  PC  while  the  graphics  workstation  Is  used  for  3D  visual  display.  The  Phantom 
is  both  an  input  device  and  an  output  device.  It  can  apply  forces  back  to  the  user  along  the  three  translational 
axes.  The  position  and  orientation  of  the  tool  or  probe  mounted  to  its  tip  are  measured  using  encoders. 

;•  trumented  surgical  tools  or  mechanical  tools  mounted  to  the  Phantom  provide  a  familiar  interface  to  the 
user.  A  powered,  two  finger  device  has  been  designed  by  the  MIT  group  and  tested  at  BDI.  This  device  can 
apply  a  gripping  torque  to  the  user  to  simulate  the  sensation  of  grasping  an  object  with  two  fingers.  Further 
development  of  the  haptic  device  is  underway  and  will  include  piezo-vibration  stimulators  for  ultra-high 
bandwidth  sensation,  multi-fingertip  interaction,  and  large  workspace. 

Haptic  Algorithms 

Haptic  algorithms  mediate  the  exchange  of  forces  between  simulated  objects  and  the  user.  The  two  most 
important  haptic  algorithms  are  for  contact  detection  and  contact  force  modeling.  Contact  detection  is  used 
for  determining  when  two  simulated  objects  are  touching.  Contact  force  models  are  used  to  compute  the 
forces  of  interaction  between  simulated  objects.  Contact  forces  can  arise  from  interpenetration  of  objects, 
sliding  between  objects,  or  from  haptic  textures  applied  to  surface  geometry. 


Contact  Detection 

The  contact  detection  algorithm  determines  when  simulated  objects  collide  or  when  the  user  is  touching  a 
simulated  object.  The  algorithms  use  the  geometry  of  the  simulated  objects  to  compute  the  distance 
separating  them.  We  use  severa1  contact  detection  algorithms  that  are  optimized  for  real-time  performance 
and  that  are  tuned  to  the  object  model  type.  For  example,  we  use  a  linear-time  Lin  and  Canny  type  algorithm 
for  contact  between  convex  rigid  polyhedra.  This  general  contact  algorithm  is  insensitive  to  the  number  of 
polygons  on  an  object.  We  use  a  grid-depth  type  algorithm  for  point  contact  on  polyhedra  with  mild 
concavities  and  a  specialized  algorithm  for  computing  contact  between  points  or  lines  and  a  flexible  tube. 


Contact  Force  Models 

The  interaction  force  between  two  touching  simulated  objects  is  computed  with  a  contact  force  model.  We 
use  penalty  method  techniques  in  our  contact  force  models.  Characteristics  such  as  surface  compliance, 
friction,  textures,  puncturability,  and  surface  feature  resolution  are  controlled  by  the  contact  force  model  and 
can  be  modified  to  suit  the  application. 
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Haptic  Texture  Mapping 


The  goal  of  haptic  texture  mapping  is  to  permit  hapticuily  perceivable  textures  to  be  overlaid  on  ba*;ic  object 
geometry.  A  haptic  texture  is  a  force  model  whose  parameters  depend  upon  the  spatial  location  of  the  haptic 
device  relative  to  the  underlying  geometry.  Our  basic  technique  is  to  superpose  forces  due  to  haptic  textures 
and  forces  due  to  contact  with  the  underlying  geometry.  The  spatial  information  for  a  haptic  texture  is 
derived  from  loadable  images  and  can  be  used  to  change  different  parameters  in  the  contact  force  models. 
Using  haptic  textures  we  have  created  the  feeling  of  scratchy  sand  paper,  smooth  bumps,  and  discrete  i  idges 
on  a  flat  surface.  We  are  currently  working  on  mapping  haptic  textures  to  rigid  objects  and  deformable  organ 
models. 

Physics-Based  Simulation 

Our  system  uses  real-time  physics-based  simulation  to  calculate  the  behavior  of  objects,  interactions  among 
objects,  and  interactions  between  objects  and  the  user.  Objects  can  be  isolated  bodies,  mechanical  linkages, 
or  more  complicated  things  like  tool  mechanisms,  deformable  tissues,  or  organs.  A  basic  idea  of  the  work  is 
that  physics-based  simulation  allows  virtual  objects  to  move  as  real  objects  do;  obeying  the  laws  of  physics, 
they  feel  like  real  objects  when  touched,  and  move  realistically  in  response  to  touching. 

The  foundation  of  our  object  simulator  is  sc  rtware  that  we  developed  in  previous  work  that  automatically 
creates  highly-optimized  dynamic  simulations  of  things  that  move  from  simple  descriptions  of  the  objects. 
The  phy  sical  simulations  include  material  properties  than  can  be  changed  by  the  user  to  tune  the  feel  of  the 
object.  For  example,  we  can  change  the  mass  of  a  rigid  body,  the  compliance  of  its  surface,  or  the  coefficient 
of  friction  between  a  surface  and  the  haptic  device. 


Rigid  Object  Loader 

A  general  object  loader  is  working  that  allows  multiple  3D  objects  to  be  loaded,  dynamically  simulated,  and 
touched  or  manipulated.  The  software  includes  automatic  contact  detection,  equation  generation,  and  3D 
graphics.  The  shape  of  these  objects  can  be  defined  with  a  CAD  file  that  describes  their  surface  geometry. 
The  geometry  is  used  for  both  visualization  and  contact  detection.  Bulk  properties  of  the  bodies  like  mass, 
coefficient  of  friction,  or  surface  impedance  can  be  specified  by  the  user.  The  haptic  tool  used  for 
manipulation  in  the  virtual  environment  is  a  special  kind  of  simulated  object.  It's  position  and  orientation  are 
driven  by  the  Phantom.  The  forces  of  interaction  between  it  and  other  simulated  objects  are  computed  from 
the  haptic  algorithms. 


Flexible  Tissue 

To  support  the  simulation  of  surgical  procedures,  we  have  expanded  our  simulation  library  to  include 
prototype  simulations  of  flexible  tissue  and  flexible  tubes.  At  BDI  we  are  using  flexible  tissue  simulation 
techniques  that  strive  for  computational  simplicity  and  speed  while  providing  convincing  behavior.  One 
prototype  flexible  tissue  simulation  included  a  planar  surface  that  deformed  when  pushed  or  pulled.  Rigid 
objects  with  varying  surface  compliance  could  be  positioned  beneath  the  flexible  tissue  to  simulate  a 
palpation  task.  We  also  have  simulations  of  whole  and  severed  hollow  tubes.  The  tubes  move  when  touched 
and  the  surfaces  deform  when  the  tubes  are  poked  or  plucked.  By  varying  the  radii  of  the  tubes  with  time  we 
can  simulate  the  look  and  feel  of  a  pulsing  vessel.  The  flexibility  or  surface  deformation  of  a  flexible  tube 
can  be  tuned  to  best  approximate  the  fee!  of  biological  material. 
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Commercialization  and  Related  Work 


BDI  is  developing  and  selling  products  that  are  closely  related  to  the  ARJPA  funded  work. 

•  BDI  is  marketing  Tangible  Reality(TM),  a  turn-key  virtual  environment  system  that  includes  visual 
images,  sound,  and  touch.  The  system  consists  of  a  force-feedback  device,  a  real-time  simulation 
computer,  a  physics-based  simulation  environment,  simulated  objects,  real-time  3Dcomputer 
graphics,  real-time  3D  sound  effects,  and  networking.  Early  versions  of  the  product  will  support 
standard  sets  of  physics-based  simulated  objects.  Later  versions  will  support  user-defined  objects 
and  deformable  objects.  This  product  provides  an  initial  channel  for  commercialization  of  results 
from  the  proposed  research.  For  example  "palpating  in"  and  "haptic  texture  mapping"  are  obvious 
candidates  for  enhancing  the  product  as  we  understand  and  develop  them. 

•  In  a  joint  project  with  Musculographics  Inc.,  we  have  delivered  the  beta  version  of  a  limb  trauma 
simulator  to  the  US  Army  at  Fort  Bragg.  BDI  was  responsible  for  the  haptic  components  on  this 
project.  Three  additional  simulators  will  be  delivered  this  year. 

•  We  won  a  competitive  bid  to  develop  a  knee  anthroscopy  simulator  for  the  American  Board  of 
Orthropedic  Surgeons.  We  will  demonstrate  this  simulator  to  the  Board  in  February.  They  are 
enthusiastic  about  possible  commercial  applications  of  the  simulator  for  testing  and  accreditation. 

•  Demonstrated  the  surgical  simulator  along  with  the  performance  assessment  software  at  the  Society 
of  Vascular  Surgeons  conference  in  January  1997  and  the  American  College  Surgeons  conference  in 
October  1997. 

•  We  delivered  a  special  version  of  Tangible  Reality  designed  for  training  surgical  procedures.  This 
version  of  the  system  supports  two-handed  anastomosis.  Thomas  >  ;  Krummel,  M.D.,  Chair  of  the 
Department  of  Surgery  at  Penn  State's  Hershey  Medical  Center  is  building  "The  Virtual  Hospital." 
The  VR  Anastomosis  Simulator  is  a  key  feature  in  his  concept  for  reducing  the  cost  and  improving 
the  quality  of  medical  teaching. 

•  We  delivered  to  Digital  Media  Associates  a  system  that  allows  the  user  to  touch,  poke  and  pluck  a 
simulated,  beating  heart  with  medical  instruments.  The  user  can  touch  a  3D  volumetric  display  of 
the  heart  with  medical  instruments  mounted  to  Phantom  force-feedback  devices.  The  user  actually 
feels  the  heart  whenever  the  physical  tool  "touches"  the  floating  3D  image  because  the  visual  and 
haptic  images  of  the  heart  are  coincident  ir.  space.  An  audible  heart  beat  is  synchronized  with  the 
visual  and  haptic  heart  displays.  We  have  plans  to  continue  this  joint  project  to  combine  force 
feedback  haptics,  dynamic  simulation,  and  3D  volumetric  displays. 

•  Nissho  Electronics  Corp.  is  a  Japanese  distributor  of  the  BDI  Tangible  Reality  System. 

•  In  an  effort  to  demonstrate  the  applicability  of  VR  technology  to  aircraft  maintenance  training  we 
developed  the  VR  Maintenance  Trainer  for  the  Joint  Advanced  Strike  Technology  Program.  This 
project  significantly  leveraged  our  ARPA-funded  project  and  offers  other  military  applications. 
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Conclusions 


As  part  of  the  "Look  and  Feel"  project  we  have  assembled  much  of  the  basic  technology  required  to  build 
integrated  haptics  systems  that  allow  the  user  to  see,  hear,  and  touch  simulated  objects.  These  worlds  can 
include  rigid  body,  multi-link,  aynamic  objects,  deformable  organs,  and  articulated  tools  that  can  be 
manipulated  by  the  user.  Haptic  algorithms  detect  collisions  between  simulated  objects  and  compute  contact 
forces  between  them.  Force-feedback  devices  convey  the  forces  of  interaction  to  the  user.  Physics-based 
simulation  computes  the  response  of  simulated  objects  to  external  forces  and  3D  computer  graphics  display 
the  visual  scene  to  the  user.  Using  these  component  technologies  we  have  constructed  prototypical  advanced 
haptic  systems  that  allow  the  user  to  perform  procedures  such  as  surgical  anastomosis,  heart  palpation,  and 
aircraft  maintenance.  While  these  prototype  demonstrations  do  not  replicate  all  the  detail  found  in  real 
surgeries  or  training  environments,  we  believe  they  take  a  significant  step  towards  the  goal  of  building  useful 
virtual  training  environments.  In  the  future,  we  will  work  towards  commercialization  of  this  technology. 
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A  NOVEL  VIRTUAL  REALITY  SURGICAL  TRAINER  WITH  FORCE  FEEDBACK: 
SURGEON  VS.  MEDICAL  STUDENT  PERFORMANCE 

O'Toole,  R.1*2,  Playter,  R.1,  Blank,  W. ',  Cornelius,  N.1,  Roberts,  W.1,  and  Raibert,  M.1 
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INTRODUCTION:  We  have  developed  an  interactive  virtual  reality  (VR)  surgical  trainer  with 
force-feedback  to  simulate  the  anastomosis  of  vessels  (See  Figure  1).  The  VR  surgical  trainer 
assesses  skill  level  during  a  simulated  surgical  procedure.  The  virtual  surgical  performance  is 
evaluated  based  upon  measures  of  damage  to  the  tissue,  needle  technique,  and  other  dynamic 
parameters.  An  initial  experiment  has  compared  the  performance  of  surgeons  and  medical 
students  on  a  simulated  surgical  task.  We  found  that  on  average  the  surgeons  performed  better 
than  the  medical  students. 


Figure  I .  The  virtual  reality  surgical  trainer  lets  you  see,  touch,  and  feel  simulated  organs  using 
3D  computer  graphics  and  advanced  force  feedback  technology.  Using  a  needle  holder  and  forceps 
attached  to  force  feedback  devices,  the  user  can  grasp,  poke,  pluck,  and  suture  flexible  tube  organs.  The 
user  can  feel  the  vessels  when  touched  and  the  vessels  move  realistically  in  response  to  touch.  The 
surgical  station  uses  a  mirror  arrangement  to  place  the  3D  image  of  the  patient  in  the  correct  position 
relative  to  the  user.  This  system  allows  the  user  to  control  a  virtual  needle  and  thread  to  perform  a 
simulated  end-to-end  anastomosis. 
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CLINICAL  RELEVANCE:  Virtual  reality  simulations  hold  the  potential  to  improve  surgical 
training  in  a  manner  analogous  to  its  impact  on  the  training  of  pilots.  The  advantages  of  VR  for 
surgical  training  include  a  reduction  in  operating  room  time  required  for  training,  the 
quantification  of  trainees’  performances,  and  an  ability  to  provide  experiences  independent  of  a 
hospital's  limited  patient  population.  An  interactive  virtual  reality  surgical  trainer  could  improve 
the  training  of  surgeons  and  also  play  a  role  »n  the  accreditation  process. 

METHODS:  We  conducted  an  experiment  to  compare  the  performance  of  medical  students  and 
vascular  surgeons  using  the  VR  surgical  trainer.  Both  groups  were  given  the  task  of  repeatedly 
inserting  a  curved  needle  through  a  simulated  vessel.  Twelve  Harvard  Medical  School  students 
and  nine  vascular  surgeons  from  five  Boston  area  hospitals  participated  in  the  study.  We  tracked 
eight  parameters  during  the  study:  1)  tissue  damage,  2)  accuracy,  3)  peak  force  applied  to  the 
tissue,  4)  time  to  complete  the  task,  5)  surface  damage  to  the  tissue,  6)  total  tool  movement,  7) 
average  angular  error  from  an  ideal  needle  path,  and  8)  an  overall  error  score.  The  users  were 
given  visual  feedback  regarding  their  performance  after  each  trial  of  the  experiment. 

RESULTS:  The  surgeons  tended  to  outperform  the  medical  students  by  having  less  tissue 
damage  (See  for  example  Figure  2).  The  surgeons'  average  performance  scores  were  better  than 
those  of  the  medical  students  for  all  of  the  measured  parameters  (See  Figure  3).  These 
differences  were  statistically  significant  (p<0.05)  for:  damage  to  the  tissue,  time  to  complete  the 
task,  excessive  tool  motions,  proper  angle  of  the  needle,  and  overall  error  score. 


Average  Tissue  Damage  vs.  Trial  Number 


Figure  2.  The  average  tissue  damage  values  are  plotted  versus  the  trial  number  in  the  experiment.  Tissue 
damage  is  defmed  as  the  sum  of  forces  that  exceed  a  tissue  damage  threshold. 
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Figure  3.  The  bar  chart  displays  the  average  medical  student  and  surgeon  performances  for  the  simulated  suigical 
task.  The  values  have  been  normalized  to  the  average  surgeon  values.  The  surgeon  values  were  better  than  the 
medical  student  values  for  all  eight  of  the  metrics.  Five  of  the  differences  (performance  error  score,  tissue  damage, 
excess  time,  excess  tool  motion,  and  angle  error)  were  statistically  significant  (p<0.05). 

CONCLUSIONS:  We  have  developed  a  VR  trainer  that  tests  a  user’s  technique  in  placing  a 
needle  through  a  simulated  vessel.  An  initial  experiment  has  compared  the  simulated  surgical 
performance  of  medical  students  and  vascular  surgeons.  On  average,  the  surgeons  outperformed 
the  medical  students  in  the  8  parameters  that  were  measured  (statistically  significant  for  5  of  the 
parameters)  .  These  results  suggest  that  the  VR  surgical  trainer  may  be  useful  in  quantifying 
surgical  skill. 
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Summary.  Simulators  based  on  virtual  reality  offer  great  promise  for  training 
future  surgeons.  We  are  developing  a  surgical  simulator  with  emphasis  on  pro¬ 
viding  the  user  with  a  sense  of  touch  in  addition  to  the  senses  of  sight  and  sound. 
Users  hold  real  medical  instruments  and  touch,  grasp,  and  suture  two  simulated 
tube  organs  as  they  practice  end-to-end  anastomosis  procedures.  The  medical 
instruments  are  attached  to  force  feedback  devices  that  convey  the  forces  of 
interaction  to  the  user.  The  behaviour  of  the  tubes  is  derived  from  a  physics- 
based  simulation.  Computer  graphics  provide  real-time  texture-mapped  images 
of  the  surgical  scene.  The  system  is  currently  operational  and  its  training  value 
will  be  evaluated  using  medical  students  and  experienced  surgeons. 

Keywords:  surgical  simulation,  haptic  feedback,  force  feedback,  virtual  reality 


Introduction 

Someday  surgical  simulators  based  on  virtual  reality 
will  allow  surgical  training  without  the  use  of  human 
patients,  hospital  space  or  animals.  Surgical  simula¬ 
tors  will  train  sensory-motor  skills,  perceptual  tasks 
and  cognitive  decision-making,  all  in  the  context  of  per¬ 
forming  relevant  surgical  procedures.  In  addition  to 
allowing  medical  students  to  practice  routine  pro¬ 
cedures,  surgical  simulators  will  allow  students  to 
encounter  patients  with  rare  medical  conditions  or 
unexpected  complications  and  practice  techniques  to 
handle  these  situations.  Using  today's  opportunistic 
methods,  only  very  experienced  surgeons  are  exposed 
to  the  wide  range  of  conditions  and  complications  that 
can  occur.  In  addition  to  direct  training,  surgical  simu¬ 
lators  could  be  used  to  assess  surgical  performance 
and  play  a  role  in  certification. 

Computer-based  surgical  simulation  has  been  under 
development  for  several  years  [1],  The  first  simulator  to 
use  virtual  medical  instruments  [2]  allowed  the  user  to 
visually  tour  an  abdomen  populated  with  simplified  organs. 
More  recent  work  in  neurosurgery  [3, 4]  and  endoscopy  [5], 
focused  on  increasing  visual  realism  by  using  data  from 
real  patients.  Other  groups  have  increased  the  level  of 
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interactivity  with  the  virtual  model  by  using  passive  input 
devices  to  ‘grab’  and  move  anatomical  parts  [6, 7]. 

The  use  of  physics-based  models  has  improved  the 
realism  and  predictive  capability  of  surgical  simulators 
for  pre-operative  planning  [8,  9],  By  using  real  surgical 
tools  as  input  devices,  minimally  invasive  surgical  simula¬ 
tors  approximate  actual  surgeries  when  interacting  with 
the  virtual  model  [10].  Force  feedback  is  being  used  in  a 
laparoscopic  surgery  trainer  [11,12]  and  lumbar  punctures 
[13,  14], 

Our  focus  in  surgical  simulation  is  to  give  the  user  a 
sense  of  touch,  in  addition  to  the  senses  of  sight  and  sound. 
Users  are  able  to  reach  into  the  virtual  environment  with 
their  hands  to  touch,  feel  and  manipulate  simulated  organs, 
in  addition  to  seeing  and  hearing  them.  Interaction  through 
touch  and  manipulation  is  known  as  haptic  interaction. 
Haptic  interaction  plays  a  prominent  role  in  the  surgical 
trainer  we  are  building. 

Virtual  anastomosis 

The  surgical  simulator  we  are  building  focuses  on  open 
surgery  anastomosis,  the  task  of  suturing  tube-like 
organs  together.  Anastomosis  is  ubiquitous  in  surgery, 
applying  to  blood  vessels,  oesophagus,  ureters,  bronchi, 
colon,  ducts  and  other  organ  systems  found  throughout 
the  human  body.  The  Virtual  Surgery  Simulator  (Figures 
1  and  2)  allows  the  user  to  practice  the  elements  of  an 
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Figure  1.  The  Virtual  Surgery  Simulator  uses  a  mirror  to  overlay 
visual  imagery  from  the  virtual  work  space  onto  the  surgeon's 
hands. 


end-to-end  anastomosis  including  touching,  grasping 
and  suturing  two  compliant  tube  organs  with  forceps, 
needle  and  needle  driver.  By  combining  visual,  audio 
and  haptic  displays,  this  trainer  allows  the  user  to  work 
on  simulated  organs  In  an  Intuitive  way  while  using 
familiar  medical  instruments. 

Thu  hardware  (Figure  3)  used  in  the  surgical  simulator 
Includes  two  Phantom  force  feedback  devices  (manu¬ 
factured  by  SensAble  Technologies,  Cambridge,  MA)  that 
provide  high  fidelity  force  feedback,  a  high-end  PC  that 
simulates  virtual  objects  and  computes  haptic  algorithms, 
and  a  Silicon  Graphics  Maximum  High  Impact  Indigo  2 
workstation  that  provides  3-D  graphics  and  sound. 

The  user  works  through  a  set  of  real  surgical  tools  that 
have  been  connected  to  force  feedback  devices.  Holding 
a  needle  holder  in  one  hand  and  a  forceps  in  the  other, 
the  user  can  grasp  and  stabilize  a  tube  while  puncturing 
it  with  a  needle  held  in  the  jaws  of  the  needle  holder. 
The  user  sutures  the  tubes  together  by  puncturing  each 
tube  in  sequence,  then  pulling  the  suture  material  tight. 
The  forces  of  interaction  between  the  tools  and  the  simu¬ 
lated  tubes  are  calculated  by  a  simulation  computer, 
then  conveyed  to  the  user  through  the  force  feedback 
devices.  Visual  images  of  the  interaction  are  displayed 
through  a  3-D  graphics  computer.  Coordinated  visual 
and  tactile  displays  allow  the  user  to  work  in  an  intuitive 
way  while  exploring  the  basic  techniques  of  the  surgical 
anastomosis. 

The  Surgery  Simulator  has  subsystems  for  real-time  3-D 
computer  graphics,  physics-based  simulation  and  haptic 


Figure  2.  The  Virtual  Surgery  Simulator:  view 
from  the  surgeon’s  perspective. 
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Figure  3.  Hardware  configuration  for  the  surgical  simulator. 


interaction.  The  quality  of  the  interaction  depends  on  how 
realistically  the  haptic  computer  simulates  the  virtual  objects, 
how  well  the  haptic  algorithms  compute  contact  forces,  how 
well  the  force  feedback  device  conveys  those  forces  back  to 
the  user,  and  how  well  the  haptic  feedback  synchronizes  the 
visual  and  auditory  feedback  to  the  user.  The  immersive 
quality  of  the  simulator  depends  on  both  the  speed  of  the  dis¬ 
plays  and  the  realism  of  interaction. 

The  computer  graphics  are  provided  by  a  conventional 
system:  it  generates  stereo,  texture-mapped  images  using 
custom  OpenGL  software  running  at  20  Hz  on  a  Silicon 
Graphics  Maximum  Impact.  The  visual  display  of  the  simu¬ 
lator  includes  two  tubes,  a  background  and  models  of  the 
surgical  tools  (needle,  suture,  needle  holder  and  forceps). 
The  polygon  count  of  the  entire  scene  with  two  tubes  is 
about  1200  polygons.  Images  are  displayed  through  a  mir¬ 
ror  system  that  places  the  visual  workspace  in  front  of  the 
user  at  about  waist  level.  This  arrangement  allows  the 
visuai  workspace  and  haptic  workspaces  to  overlap,  result¬ 
ing  in  a  natural  working  configuration  (Figures  1  and  2).  A 
hand  rest  allows  the  surgeon  to  stabilize  his  or  her  hands 
for  delicate  manipulations. 

The  software  for  the  system  includes  several  components: 

•  Deformable  tube  simulation, 

•  Fast  contact  detection, 
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•  Contact  force  calculation, 

•  Kinematics  for  feedback  devices, 

•  Force  feedback  avice  drivers, 

•  3-D  graphics  and  sound  on  SGI  computers. 

Physics-based  simulation  is  used  to  model  the  deformable 
tubes  in  the  system  and  compute  their  response  to  inter¬ 
actions  with  the  surgical  tools.  We  designed  a  specialized 
tube  simulator  that  has  good  real-time  performance  at  the 
expense  of  generality.  The  tube  simulator  uses  a  lumped 
mass  model  for  motions  of  the  free  end,  an  elastic  spine 
for  tube  bending  and  stretching,  and  focal  surface  deforma¬ 
tion  models  that  permit  puckering  where  the  tube  is 
pinched,  and  dimpling  where  it  is  poked.  Physical  para¬ 
meters  of  the  tubes  such  as  the  diameter,  thickness,  length, 
compliance,  mass,  surface  friction  and  resistance  to  punc¬ 
ture  can  be  modified  to  represent  different  anatomical 
elements.  Rather  than  measure  the  elasticity  of  biological 
skin,  we  asked  practicing  surgeons  to  evaluate  the  feel  of 
the  simulated  tissue  in  our  surgical  station.  We  adjusted 
parameters  of  the  tissue  model  until  the  physicians  felt  it 
was  realistic.  The  simplified  tube  simulation  does  not  yet 
support  layers  of  tissue,  tissue  folding  or  contact  against 
puckered  or  dimpled  regions  of  the  tube. 

Haptic  algorithms  mediate  the  exchange  of  forces 
between  simulated  objects  and  the  user.  The  two  most 
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important  haptic  algorithms  are  those  that  detect  when 
objects  contact  one  another,  and  that  compute  the  forces 
created  when  contact  occurs.  We  use  several  contact 
detection  algorithms  that  are  optimized  for  real-time  perfor¬ 
mance  and  that  are  tuned  to  the  object  model  type.  For 
example,  we  use  a  linear-time  Lin  and  Canny  [15]  type 
algorithm  for  contact  between  convex  rigid  polyhedra.  For 
point  contact  on  static  polyhedra  with  mild  concavities  we 
store  the  polyhedra  in  memory  as  a  series  of  'depth'  values, 
then  compare  the  depth  of  the  tool  point  to  that  of  the  poly¬ 
hedra  .  To  calculate  contact  forces  between  objects,  we  use 
penalty  method  techniques  that  calculate  contact  force 
from  spring  and  damper  models  and  the  interpenetration 
of  the  two  objects.  Characteristics  such  as  surface  compli¬ 
ance,  friction,  texture,  puncturability  and  surface  feature 
resolution  are  controlled  by  the  contact  force  model  and 
can  be  modified  to  suit  the  application. 

The  haptic  display  rate  required  for  a  good  quality  sense 
of  touch  depends  upon  the  objects  being  simulated.  Con¬ 
vincing  interactions  with  rigid  objects  generally  require 
1000  Hz  while  interactions  with  more  compliant  objects, 
like  soft  tissue,  ore  less  demanding.  The  surgery  simulator 
has  an  update  rate  for  the  haptic  algorithms  and  simulation 
of  about  400  Hz.  To  achieve  this  rate  we  limited  the  number 
of  possible  contacts  between  different  objects  in  the  scene. 
For  example,  the  tools  cannot  contact  one  another  and  the 
needle  can  only  contact  the  tube  at  its  tip  until  puncture. 
After  puncture,  the  contact  point  travels  along  the  needle 
shaft  so  that  the  tissue  ‘holds'  the  needle  at  the  proper 
point. 

Discussion 

The  goal  of  this  work  is  to  build  a  surgical  simulator  that  can 
be  used  for  training  surgical  procedures  and  evaluating  sur¬ 
gical  performance.  It  is  unlikely  that  all  the  detail  found  in 
real  surgeries  will  be  captured  by  surgical  simulators  in 
the  near  term.  However,  it  is  feasible  to  capture  enough 
of  the  real  experience  to  provide  useful  training  of  surgical 
procedures  and  performance  evaluation.  Despite  Its  many 
limitations,  we  believe  the  surgical  simulator  takes  a  sig¬ 
nificant  step  towards  this  realistic  goal. 

Our  current  work  involves  expanding  the  capabilities  and 
improving  the  performance  of  the  surgical  simulator.  We 
are  adding  performance  measurement  and  display  fea¬ 
tures  to  the  simulator.  These  recording,  display  and  play¬ 
back  features  will  help  the  surgical  trainee  review  and 
improve  surgical  technique  by  providing  quantitative  mea¬ 
sures  of  needle  placement  accuracy,  force  level  and  tissue 
damage.  Once  these  features  are  working,  the  user  will  be 
able  to  practice  important  elements  of  surgical  anastomo¬ 
sis,  including  proper  handling  of  the  medical  tools, 


techniques  for  grasping  tubes  and  suturing  techniques 
including  proper  needle  orientation,  needle  placement, 
puncture  forces,  suture  spacing  and  suture  tightening. 
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